We will describe, in the main, signal processors which employ neural networks and other techniques to generate output data examples which match those previously learnt. For example the signal processor may be trained with many different examples of handwritten digits from zero to nine and may then be employed to randomly generate a new example from one of the learnt categories. Thus an output may be generated from a set of learnt distributions (of the training examples) and, in general, the categorisation of the training examples may also be learnt. We will also describe techniques which use an external input to select the category of output example generated, not by precisely specifying the category but instead by providing data which defines a ‘context’ for the training examples. The signal processor is trained using examples and their context and then afterwards context data can be used to bias the generation of output examples.
Signal processors of this general type have a range of applications. For example they can be used for prediction, with or without context, and thus have applications in many types of image and audio signal processing, as well as in control applications, for example predicting the position of a robot arm, as well as in other applications, for example evolutionary search techniques for, say, drug discovery. Embodiments of the signal processor/system may process data including, but not limited to: audio, video, image, game, sensor, actuator, control (including motor control), biological, physical, chemical, spatial, text, search, and other data.
It is known to use a Boltzmann machine to provide a so-called generative model as described, for example, in Salakhutdinov and Hinton, “Deep Boltzmann Machine”, in Proceedings of the International Conference on Artificial Intelligence and Statistics, volume 5, pages 448-455, 2009. However Deep Boltzman Machines require a great deal of processing power to implement.
A Helmholtz machine can also be employed to provide a generative model, but whilst such machines have some interesting features in practice they learn very slowly and the output examples they generate are poor.
We will describe improved signal processors and related architectures which address both these problems.